Constraint Automata and the Complexity of Recursive Subtype Entailment

We study entailment of structural and nonstructural recursive subtyping constraints. Constraints are formal inequalities between type expressions, interpreted over an ordered set of possibly infinite labeled trees. The nonstructural ordering on trees is the one introduced by Amadio and Cardelli for subtyping with recursive types. The structural ordering compares only trees with common shape. A constraint set entails an inequality if every assignment of meanings (trees) to type expressions that satisfies all the constraints also satisfies the inequality. In this paper we prove that nonstructural subtype entailment is PSPACEhard, both for finite trees (simple types) and infinite trees (recursive types). For the structural ordering we prove that subtype entailment over infinite trees is PSPACE-complete, when the order on trees is generated from a lattice of type constants. Since structural subtype entailment over finite trees has been shown to be coNP-complete these are the first complexity-theoretic separation results that show that, informally, nonstructural subtype entailment is harder than structural entailment, and recursive entailment is harder than nonrecursive entailment. 1 I n t r o d u c t i o n 1.1 C o n s t r a i n t e n t a i l m e n t Notions of entailment for subtype inequality constraints and set inclusion constraints are currently receiving much attention. Recent work in subtyping systems and in set constraints using notions of constraint entailment in the sense of the present paper includes [1, 5, 15, 22, 6, 2, 8, 13]. In subtyping systems, a program is typed under a set of suptyping constraints consisting of inequalities of the form T < T ~, where T and T ~ are type expressions; the constraints express hypotheses about subtype relations which must hold between the types for the program to be well-typed. If C is a constraint set and r is an inequality (such as a subtype inequality or a set inclusion), we say that C entails r written C ~ r if every assignment of meanings to expressions that satisfies all the constraints in C also satisfies r an inequality is satisfied under the assignment, if it is true